Systems and methods for onscreen identification of instruments in a teleoperational medical system

ABSTRACT

A teleoperational assembly is disclosed which includes an operator control system and a plurality of manipulators configured to control the movement of medical instruments in a surgical environment. The manipulators are teleoperationally controlled by the operator control system. The system further includes a processing unit configured to display an image of a field of view of the surgical environment, determine association information about the manipulators and the operator control system, and display badges near the medical instruments in the image of the field of view of the surgical environment. The badges display the association information for the medical instrument they appear associated with.

RELATED APPLICATIONS

This patent application is the U.S. national phase of InternationalApplication No. PCT/US2016/022612, filed Mar. 16, 2016, which designatedthe U.S. and claims priority to and the benefit of the filing date ofU.S. Provisional Patent Application 62/134,297, entitled “SYSTEMS ANDMETHODS FOR ONSCREEN IDENTIFICATION OF INSTRUMENTS IN A TELEOPERATIONALMEDICAL SYSTEM,” filed Mar. 17, 2015, which are incorporated byreference herein in their entireties.

FIELD

The present disclosure is directed to systems and methods for performinga teleoperational medical procedure and more particularly to systems andmethods for displaying information about teleoperational instrumentsused in a surgical environment.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during invasive medical procedures, therebyreducing patient recovery time, discomfort, and harmful side effects.Such minimally invasive techniques may be performed through naturalorifices in a patient anatomy or through one or more surgical incisions.Through these natural orifices or incisions, clinicians may insertmedical tools to reach a target tissue location. Minimally invasivemedical tools include instruments such as therapeutic instruments,diagnostic instruments, and surgical instruments. Minimally invasivemedical tools may also include imaging instruments such as endoscopicinstruments. Imaging instruments provide a user with a field of viewwithin the patient anatomy. Some minimally invasive medical tools andimaging instruments may be teleoperated or otherwise computer-assisted.In a teleoperational medical system, instruments may be difficult tosee, whether due to small instrument size, proximity to the edge of thefield of view, or obstruction by tissue. A clinician operating theteleoperational system may also have difficulty keeping track of whichinstruments are under which operator controls. Systems and methods areneeded to provide a clinician with information about the instrumentsunder the clinician's control to reduce the risk of clinician error.

SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

In one embodiment, a teleoperational assembly includes an operatorcontrol system and a plurality of manipulators configured to control themovement of medical instruments in a surgical environment. Themanipulators are teleoperationally controlled by the operator controlsystem. The system further includes a processing unit configured todisplay an image of a field of view of the surgical environment,determine association information about the manipulators and theoperator control system, and display badges near the medical instrumentsin the image of the field of view of the surgical environment. Thebadges display the association information for the medical instrumentthey appear associated with.

In another embodiment, an image of a field of view of a surgicalenvironment is displayed. The locations of medical instruments withinthe environments are determined, association information for the medicalinstruments and for an operator control system is determined, and badgesare displayed in the image of the field of view of the surgicalenvironment near the medical instruments. The medical instruments areattached to manipulators that are teleoperationally controlled by theoperator control system. The badges display the association informationfor the medical instrument near which they are displayed.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, thedimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1A is a schematic view of a teleoperational medical system, inaccordance with an embodiment of the present disclosure.

FIG. 1B is a perspective view of a surgeon's control console for ateleoperational medical system, in accordance with many embodiments.

FIG. 1C is a perspective view of a teleoperational medical systemelectronics cart, in accordance with many embodiments.

FIG. 1D is a perspective view of a patient side cart, according to oneexample of principles described herein.

FIG. 2A illustrates a field of view of a surgical workspace with atleast one medical instrument visible in the field of view, theinstrument having a badge associated with it and containing associationinformation pertaining to the instrument.

FIG. 2B is a flowchart illustrating a method for operating ateleoperational medical system.

FIGS. 3A-3C illustrate input control devices and user hand positionsthat correspond to association information badge arrangements as shownin FIGS. 3D-3F.

FIGS. 3D-3F illustrate various arrangements for an orbiting portion ofan association information badge.

FIGS. 4A-4B illustrate the size-changing behavior of the associationinformation badge as the associated instrument moves toward and awayfrom the imaging device.

FIGS. 5A-5B illustrate how the association information badge remainsvisible when the associated medical instrument is obstructed from view.

FIGS. 6A-6B illustrate the adjustment of the association informationbadge when medical instruments are in close proximity.

FIGS. 7A-7C illustrate the behavior of the association information badgeas the associated medical instrument nears the edge of the image of thesurgical environment, and as the medical instrument moves outside of theimage of the surgical environment.

FIG. 8 illustrates a method of rendering badges within an image of thefield of view of the surgical workspace.

FIG. 9 illustrates in detail the process of FIG. 8 wherein the locationsof medical instrument distal end portions are projected into acoordinate space of the image of the field of view of the surgicalworkspace.

FIG. 10 illustrates in detail the process of FIG. 8 wherein potentialoverlap of badges is checked for and resolved. In this case there are atleast two medical instruments.

FIG. 11 illustrates in detail the process of FIG. 8 wherein badges areconstrained to the viewable space of the image of the field of view ofthe surgical workspace.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the disclosure is intended. In the following detaileddescription of the aspects of the invention, numerous specific detailsare set forth in order to provide a thorough understanding of thedisclosed embodiments. However, it will be obvious to one skilled in theart that the embodiments of this disclosure may be practiced withoutthese specific details. In other instances well known methods,procedures, components, and circuits have not been described in detailso as not to unnecessarily obscure aspects of the embodiments of theinvention.

Any alterations and further modifications to the described devices,instruments, methods, and any further application of the principles ofthe present disclosure are fully contemplated as would normally occur toone skilled in the art to which the disclosure relates. In particular,it is fully contemplated that the features, components, and/or stepsdescribed with respect to one embodiment may be combined with thefeatures, components, and/or steps described with respect to otherembodiments of the present disclosure. In addition, dimensions providedherein are for specific examples and it is contemplated that differentsizes, dimensions, and/or ratios may be utilized to implement theconcepts of the present disclosure. To avoid needless descriptiverepetition, one or more components or actions described in accordancewith one illustrative embodiment can be used or omitted as applicablefrom other illustrative embodiments. For the sake of brevity, thenumerous iterations of these combinations will not be describedseparately. For simplicity, in some instances the same reference numbersare used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X, Y, Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom). As used herein, the term“shape” refers to a set of poses, positions, or orientations measuredalong an object.

Referring to FIG.1 A of the drawings, a teleoperational medical systemfor use in, for example, medical procedures including diagnostic,therapeutic, or surgical procedures, is generally indicated by thereference numeral 10. As will be described, the teleoperational medicalsystems of this disclosure are under the teleoperational control of asurgeon. In alternative embodiments, a teleoperational medical systemmay be under the partial control of a computer programmed to perform theprocedure or sub-procedure. In still other alternative embodiments, afully automated medical system, under the full control of a computerprogrammed to perform the procedure or sub-procedure, may be used toperform procedures or sub-procedures. As shown in FIG. 1A, theteleoperational medical system 10 generally includes a teleoperationalassembly 12 mounted to or near an operating table O on which a patient Pis positioned. The teleoperational assembly 12 may be referred to as apatient side cart. A medical instrument system 14 and an endoscopicimaging system 15 are operably coupled to the teleoperational assembly12. An operator input system 16 allows a surgeon or other type ofclinician S to view images of or representing the surgical site and tocontrol the operation of the medical instrument system 14 and/or theendoscopic imaging system 15.

The operator input system 16 may be located at a surgeon's console,which is usually located in the same room as operating table O. Itshould be understood, however, that the surgeon S can be located in adifferent room or a completely different building from the patient P.Operator input system 16 generally includes one or more input controldevice(s) for controlling the medical instrument system 14. The inputcontrol device(s) may include one or more of any number of a variety ofdevices, such as hand grips, joysticks, trackballs, data gloves,trigger-guns, hand-operated controllers, voice recognition devices,touch screens, pedals, body motion or presence sensors, eye-gazetracking devices and the like. In some embodiments, the controldevice(s) will be provided with the same degrees of freedom as themedical instruments of the teleoperational assembly to provide thesurgeon with telepresence, the perception that the control device(s) areintegral with the instruments so that the surgeon has a strong sense ofdirectly controlling instruments as if present at the surgical site. Inother embodiments, the control device(s) may have more or fewer degreesof freedom than the associated medical instruments and still provide thesurgeon with telepresence. In some embodiments, the control device(s)are manual input devices which move with six degrees of freedom, andwhich may also include an actuatable handle for actuating instruments(for example, for closing grasping jaws, applying an electricalpotential to an electrode, delivering a medicinal treatment, and thelike).

The teleoperational assembly 12 supports and manipulates the medicalinstrument system 14 while the surgeon S views the surgical site throughthe console 16. An image of the surgical site can be obtained by theendoscopic imaging system 15, such as a stereoscopic endoscope, whichcan be manipulated by the teleoperational assembly 12 to orient theendoscope 15. Optionally, an electronics cart 18 can be used to processthe images of the surgical site for subsequent display to the surgeon Sthrough the surgeon's console 16. The number of medical instrumentsystems 14 used at one time will generally depend on the diagnostic orsurgical procedure and the space constraints within the operating roomamong other factors. The teleoperational assembly 12 may include akinematic structure of one or more non-servo controlled links (e.g., oneor more links that may be manually positioned and locked in place,generally referred to as a set-up structure) and a teleoperationalmanipulator. The teleoperational assembly 12 includes a plurality ofmotors that drive inputs on the medical instrument system 14. Thesemotors move in response to commands from the control system (e.g.,control system 20). The motors include drive systems which when coupledto the medical instrument system 14 may advance the medical instrumentinto a naturally or surgically created anatomical orifice. Othermotorized drive systems may move the distal end of the medicalinstrument in multiple degrees of freedom, which may include threedegrees of linear motion (e.g., linear motion along the X, Y, ZCartesian axes) and in three degrees of rotational motion (e.g.,rotation about the X, Y, Z Cartesian axes). Additionally, the motors canbe used to actuate an articulable end effector of the instrument forgrasping tissue in the jaws of a biopsy device or the like.

The teleoperational medical system 10 also includes a control system 20.The control system 20 includes at least one memory and at least oneprocessor (not shown), and typically a plurality of processors, foreffecting control between the medical instrument system 14, the operatorinput system 16, and an electronics system 18. The control system 20also includes programmed instructions (e.g., a computer-readable mediumstoring the instructions) to implement some or all of the methodsdescribed in accordance with aspects disclosed herein. While controlsystem 20 is shown as a single block in the simplified schematic of FIG.1A, the system may include two or more data processing circuits with oneportion of the processing optionally being performed on or adjacent theteleoperational assembly 12, another portion of the processing beingperformed at the operator input system 16, and the like. Any of a widevariety of centralized or distributed data processing architectures maybe employed. Similarly, the programmed instructions may be implementedas a number of separate programs or subroutines, or they may beintegrated into a number of other aspects of the teleoperational systemsdescribed herein. In one embodiment, control system 20 supports wirelesscommunication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11,DECT, and Wireless Telemetry.

In some embodiments, control system 20 may include one or more servocontrollers that receive force and/or torque feedback from the medicalinstrument system 14. Responsive to the feedback, the servo controllerstransmit signals to the operator input system 16. The servocontroller(s) may also transmit signals instructing teleoperationalassembly 12 to move the medical instrument system(s) 14 and/orendoscopic imaging system 15 which extend into an internal surgical sitewithin the patient body via openings in the body. Any suitableconventional or specialized servo controller may be used. A servocontroller may be separate from, or integrated with, teleoperationalassembly 12. In some embodiments, the servo controller andteleoperational assembly are provided as part of a teleoperational armcart positioned adjacent to the patient's body.

The teleoperational medical system 10 may further include optionaloperation and support systems (not shown) such as illumination systems,steering control systems, irrigation systems, and/or suction systems. Inalternative embodiments, the teleoperational system may include morethan one teleoperational assembly and/or more than one operator inputsystem. The exact number of manipulator assemblies will depend on thesurgical procedure and the space constraints within the operating room,among other factors. The operator input systems may be collocated, orthey may be positioned in separate locations. Multiple operator inputsystems allow more than one operator to control one or more manipulatorassemblies in various combinations.

FIG. 1B is a perspective view of the surgeon's console 16. The surgeon'sconsole 16 includes a left eye display 32 and a right eye display 34 forpresenting the surgeon S with a coordinated stereo view of the surgicalsite that enables depth perception. The console 16 further includes oneor more input control devices 36, which in turn cause theteleoperational assembly 12 to manipulate one or more instruments or theendoscopic imaging system. The input control devices 36 can provide thesame degrees of freedom as their associated instruments 14 to providethe surgeon S with telepresence, or the perception that the inputcontrol devices 36 are integral with the instruments 14 so that thesurgeon has a strong sense of directly controlling the instruments 14.To this end, position, force, and tactile feedback sensors (not shown)may be employed to transmit position, force, and tactile sensations fromthe instruments 14 back to the surgeon's hands through the input controldevices 36.

FIG. 1C is a perspective view of the electronics cart 18. Theelectronics cart 18 can be coupled with the endoscope 15 and can includea processor to process captured images for subsequent display, such asto a surgeon on the surgeon's console, or on another suitable displaylocated locally and/or remotely. For example, where a stereoscopicendoscope is used, the electronics cart 18 can process the capturedimages to present the surgeon with coordinated stereo images of thesurgical site. Such coordination can include alignment between theopposing images and can include adjusting the stereo working distance ofthe stereoscopic endoscope. As another example, image processing caninclude the use of previously determined camera calibration parametersto compensate for imaging errors of the image capture device, such asoptical aberrations. The electronics cart 18 may also include a displaymonitor and components of the control system 20.

FIG. 1D is a perspective view of one embodiment of a teleoperationalassembly 12 which may be referred to as a patient side cart. The patientside cart 12 shown provides for the manipulation of three surgical tools26 a, 26 b, 26 c (e.g., instrument systems 14) and an imaging device 28(e.g., endoscopic imaging system 15), such as a stereoscopic endoscopeused for the capture of images of the site of the procedure. The imagingdevice may transmit signals over a cable 56 to the electronics cart 18.Manipulation is provided by teleoperative mechanisms having a number ofjoints. The imaging device 28 and the surgical tools 26 a, 26 b, 26 ccan be positioned and manipulated through incisions in the patient sothat a kinematic remote center is maintained at the incision to minimizethe size of the incision. Images of the surgical site can include imagesof the distal ends of the surgical tools 26 when they are positionedwithin the field-of-view of the imaging device 28. The teleoperationalassembly 12 is located in a world coordinate system or frame F_(W). Thedistal tip of each medical instrument 26 a, 26 b, 26 c defines arespective instrument coordinate system or frame F_(I). The distal tipof the imaging device 28 defines a coordinate system or frame F_(E).

The patient side cart 12 includes a drivable base 58. The drivable base58 is connected to a telescoping column 57, which allows for adjustmentof the height of arms 54 a, 54 b, 54 c, and 54 d. Arm 54 a may be knownand/or labeled in the surgical environment as “Arm 1.” Arm 54 b may beknown and/or labeled in the surgical environment as “Arm 2.” Arm 54 cmay be known and/or labeled in the surgical environment as “Arm 3.” Arm54 d may be known and/or labeled in the surgical environment as “Arm 4.”The arms 54 a may include a rotating joint 55 that both rotates andmoves up and down. Arm 54 a connects to a manipulator arm portion 51.The manipulator arm portions 51 may connect directly to the medicalinstrument 26 a via a manipulator spar 59. Each of the other arms 54 b,54 c, 54 d may have a similar configuration to arm 54 a. Each of thearms 54 a, 54 b, 54 c, 54 d may be connected to an orienting platform53. The orienting platform 53 may be capable of 360 degrees of rotation.The patient side cart 12 may also include a telescoping horizontalcantilever 52 for moving the orienting platform 53 in a horizontaldirection. The manipulator arm portion 51 may be teleoperable. The arms54 a, 54 b, 54 c, 54 d may be teleoperable or not. In embodiments inwhich the arm 54 is not teleoperable but the manipulator arm is, the arm54 is positioned as desired before the surgeon begins operation with thesystem 10. Note that while arms 54 (and associated manipulator armportions 51) are depicted and described as being part of a singlepatient side cart 12 for examplary purposes, in various otherembodiments, arms 54 and/or manipulator arm portions 51 (or additionalarms 54 and/or manipulator arm portions 51) can be discrete structures(e.g., separate table-, ceiling-, and/or floor-mounted arms).

Endoscopic imaging systems (e.g., systems 15, 28) may be provided in avariety of configurations including rigid or flexible endoscopes. Rigidendoscopes include a rigid tube housing a relay lens system fortransmitting an image from a distal end to a proximal end of theendoscope. Flexible endoscopes transmit images using one or moreflexible optical fibers. Endoscopes may be provided with differentviewing angles including a 0° viewing angle for forward axial viewing orviewing angles between 0°-90° for forward oblique viewing. Digital imagebased endoscopes have a “chip on the tip” design in which a distaldigital sensor such as a one or more charge-coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS) device store image data.Endoscopic imaging systems may provide two- or three-dimensional imagesto the viewer. Two-dimensional images may provide limited depthperception. Three-dimensional stereo endoscopic images may provide theviewer with more accurate depth perception. Stereo endoscopicinstruments employ stereo cameras to capture stereo images of thepatient anatomy in the camera's field of view.

The stereo images of the imaging system field of view may be seen by aclinician through the eye displays 32, 34. In addition to the patienttissue, the field of view may also include the distal ends of themedical instruments and any accessories used in performing the surgicalprocedure. To perform the procedure, a clinician recognizes anassociation between an instrument in the field of view and the inputcontrol device controlling that instrument from the control console. Theclinician may make this association by, for example, referring toalphanumeric text, symbols, or other information located at theperiphery of the viewed image. To view this information, however, theclinician's focus must move away from the central portion of the displayand distal tips of the instruments to the periphery of the image.Alternatively, the clinician may make the association by moving an inputcontrol device and observing the corresponding instrument movement inthe image. This may be time consuming and disruptive to the flow of thesurgical procedure. To aid the clinician in associating the inputcontrol devices 36 with the instruments visible in the field of view,information about an input control device may be presented in graphicalform and co-located with the instrument it controls. Co-locating theassociation information with the instrument allows the clinician toquickly associate control devices with instruments while maintainingfocus on the surgical area of interest.

FIG. 2A illustrates a surgical environment 200 within the patient P. Animaging instrument (e.g., imaging instrument 28) is used to display animage 202 of the imaging instrument's field of view within the surgicalenvironment 200. The image 202 may be a three dimensional image obtainedby a stereoscopic endoscope and generated as a composite image of theimages visible to a user through right and left eye displays. The fieldof view includes a distal end portion of instrument 26 a, a distal endportion of instrument 26 b, and a distal end portion of instrument 26 c.The image 202 also includes information fields 204 located at theperiphery of the image which may include instructions to the clinician,warnings, instrument identification information, status information, orother information relevant to the surgical procedure. To assist theclinician in performing the surgical procedure using the system 10,various pieces of information are superimposed on the image 202. In thisembodiment, a badge 206 contains association information related to themedical instrument 26 b, such as a numerical identifier of the arm 54 b(e.g. Arm “3”) to which the instrument is coupled. The badge 206 isdisplayed proximate to the medical instrument 204 so that the cliniciancan easily tell that the association information pertains to the medicalinstrument 26 b. In some embodiments, the badge may also includeassociation information including the type of medical instrument, thestate of the medical instrument (e.g. cauterizer charging/ready) or thelike. Methods for creating and displaying the badge 206 are described infurther detail below, with reference to FIGS. 8-11. In variousembodiments, the badge may be circular, oval, square or any suitableshape. In various embodiments, a badge may have colors, outlines,shapes, or other features to identify or distinguish badges from oneanother.

The distal end portion of the medical instrument 26 b visible in thefield of view includes an end effector tip 208, a shaft 210, and a jointregion 212 between the shaft and end effector tip. In some embodiments,a default location for placement of the badge 206 is superimposed overthe joint region 212. In other embodiments, the default location forplacement of the badge 206 may be on the end effector tip 208 or theshaft 210. In other embodiments, the badge 206 is displayed adjacent tothe distal end portion. The default location for the badge may dependupon the size of the distal end portion of the medical instrument 26 bin the field of view. For example, if the image is closely zoomed in(e.g., FIG. 3F), and the end effector occupies a large portion of theimage, the badge may be located on a proximal portion of one of the endeffector jaws without obstructing the view at the distal end of thejaws. If, however, the image is zoomed out and the end effector jaws arerelatively small in the image, the badge may be located on the jointregion or the shaft to avoid obscuring the jaws. The placement of thebadge 206 allows the clinician to receive the association informationwhile remaining focused on the instruments and the tissue manipulated bythe instruments. The placement of the badge 206 also avoids obscuringobjects in the field of view that need to be observed by the clinician.

In some embodiments the badge 206 has a central portion 214 as well asan orbiting portion 216. The central portion 214 and orbiting portion216 may contain different pieces of association information. Forexample, as shown in FIG. 2, the central portion 214 includes a numberindicating the arm 54 b (e.g. Arm “3”) to which the medical instrument26 b is attached. In various embodiments, orbiting portion 216 containsassociation information related to the control device used by theclinician to control the medical instrument 26 b. This associationinformation may be, for example, an indication of whether a right (e.g.,“R”) or left (e.g., “L”) hand-operated input control device is currentlyin control of the medical instrument 26 b to which badge 206 isassociated.

The orbiting portion 216 of the badge 206 may move circumferentiallyrelative to the central portion 214 of badge. The circumferentiallocation of the orbiting portion 216 indicates the rotational positionof the control device, letting the clinician know where their hand onthe control device is positioned relative to the medical instrument 26 bwhich they are controlling. The function of the orbiting portion 216 isshown in greater detail in FIGS. 3A-3F.

A badge 207 is located proximate to the instrument 26 a and has similarfeatures and attributes as badge 206. Badge 207 contains associationinformation related to the medical instrument 26 a, such as a numericalidentifier of the arm 54 a (e.g. Arm “1”) to which the instrument iscoupled. In this embodiment and at this phase of the surgical procedure,instrument 26 c is not presently under control of an input controldevice. Therefore, a badge is not located proximate to the instrument 26c. In alternative embodiments, badge information may be provided for theinstrument 26 c indicating the numerical identifier of the arm 54 c(e.g., Arm “4”), but the fact that the instrument is not presently undercontrol of an input device may be indicated by color, symbol, or otherindication.

FIG. 2B illustrates a method 100 of operating a teleoperational systemto perform a surgical procedure. At a process 102, the method comprisesdisplaying an image 202 of a field of view of a surgical environment. Ata process 104, a location of a medical instrument 26 a in the surgicalenvironment is determined. At a process 106, association information forthe medical instrument 26 a is determined. The association informationincludes information about the teleoperational manipulator to which themedical instrument 26 a is coupled and/or about the operator inputcontrol that controls the medical instrument. At a process 108, a badgeis displayed proximate to an image of the medical instrument 26 a in theimage 202 of the field of view. The badge includes the associationinformation.

FIG. 3A-3C illustrates a clinician's right hand 300 controlling an inputcontrol device 302 (i.e., an input control device 36). The input controldevice 302 controls the medical instrument 26 c. As shown in FIG. 3A,from the clinician's perspective, the input control device 302 extendstoward the right. Hand association information including hand identity(right or left) and the orientation of the hand may be conveyed to theclinician via the badge. Although this control association informationis described in terms of a human hand, similar association informationmay be provided where control is delivered by foot pedals, eye gazetrackers, voice commands, or other clinician controls. As shown in FIG.3D, hand association information about the control device 302 of FIG. 3Ais conveyed visually to the clinician in the image 202 by orienting anorbiting portion 222 of a badge 218 to the right (e.g. at about a 3o′clock position) relative to the central portion 220 of the badge. Theorbiting portion 222 displays an “R” to indicate that the right-handcontrol device 302 is being used to control the medical device 26 c. Asshown in FIG. 3B, from the clinician's perspective, the input controldevice extends upward, approximately 90 degrees from the position inFIG. 3A. As shown in FIG. 3E, this orientation information about thecontrol device 302 of FIG. 3B is conveyed visually to the clinician inthe image 202 by orienting the orbiting portion 222 of the badge 218upward (e.g. at about a 12 o′clock position) relative to the centralportion 220 of the badge. As shown in FIG. 3C, from the clinician'sperspective, the input control device extends downward, approximately 90degrees from the position in FIG. 3A. As shown in FIG. 3F, thisorientation information about the control device 302 of FIG. 3C isconveyed visually to the clinician in the image 202 by orienting theorbiting portion 222 of the badge 218 downward (e.g. at about a 6o′clock position) relative to the central portion 220 of the badge.

In some alternative embodiments, hand association information may beprovided as the an illustration or 3D model of a hand. The illustrationor model portrays a left or right hand according to whether a left-handor right-hand control device is being used to control the associatedmedical device. Similar to the orbiting portion 222 in FIGS. 3D-3F, asthe orientation of the control device changes the position of theillustration or model changes accordingly to visually convey informationabout the new orientation of the control device to the clinician. Theproperties and features described for any of the badges in thisdisclosure may apply to any of the other badges in this disclosure.

Referring now to FIGS. 4A-4B, there is shown further behavior of thebadge 206 as the imaging device is moved or zoomed within the surgicalenvironment. In an embodiment shown in FIG. 4A, the imaging instrumentis moved or zoomed in toward the medical instruments 26 a, 26 b, 26 c.As the medical instruments 26 a, 26 b, 26 c appear larger in the fieldof view and the image 202, the badges 206, 207 become larger, changingsize to appear as if they are located at the same depth in the field ofview as the distal end portions of the instruments. In FIG. 4B, as theimaging instrument is moved further away or zoomed out from the medicalinstruments 26 a, 26 b, 26 c, the badges 206, 207 become smaller, againchanging to appear as if they are located at the same depth in the fieldof view as the distal end portions of the instruments. In someembodiments there is a limit on the maximum and minimum size of thebadges 206, 207 to prevent them from becoming unreadable. Further detailabout the badge positions and sizes are provided below in thedescription for FIGS. 8-11. The size of the badges 206, 207 may alsoimpart additional information to the clinician such as a distancebetween the medical instruments 26 a, 26 b and the imaging device. Thismay be helpful because different types of medical instruments vary insize, and a small instrument close to the imaging device could appear tobe the same size as a large instrument far away from the imaging device.

Since the badges 206, 207 are superimposed on the image 202, they mayappear on the display even if the view of the medical instruments 26 a,26 b is obstructed by tissue, other instruments, or the like. As shownin FIG. 5A badge 206 remains visible even as the associated instrument26 b is hidden behind tissue. The presence of the badge 206substantially offset from and at a different apparent size and depththan other instruments 26 a, 26 c visible in the field of view indicatesto the clinician that the instrument 26 b associated with the badge isoccluded. The badge 206 allows the occluded instrument 26 b to be foundas shown in FIG. 5B. In FIG. 5B the tissue has been moved and themedical instrument 26 b to which badge 206 is attached have becomevisible in the field of view.

The teleoperational medical system 10 may have multiple operationalstates for each instrument. These may include an active state, wherein amedical instrument is under the control of an input control device suchthat the clinician is actively controlling the medical instrument. In adisengaged state, a medical instrument is not under the control of aninput control device and remains stationary, locked in place by themanipulator arm to which it is coupled. The system may also have anengagement state in which the user is prompted to engage or take controlof the medical instrument via the associated input control device. Insome embodiments, the badges are displayed when the teleoperationalmedical system is in the engagement state for a particular instrument.For example, in FIG. 4A, instruments 26 a and 26 b are in an engagementstate in which the user is prompted by the badge information to takecontrol of instrument 26 a on Arm 1 with the left (“L”) input controldevice and to take control of instrument 26 b on Arm 3 with the right(“R”) input control device. Instrument 26 c is in a disengaged state andthus, a badge is not presented for the instrument. During the activestate, the badges may remain visible in the image or may be removed toavoid distraction to the clinician. Presenting the badges in theengagement state allows the clinician to recognize the associationinformation and location of each instrument within the field of view 202before taking control of any of the medical instruments. For example,the location of the badge may be used to determine if any medicalinstruments are obstructed from view by tissue before taking control ofthe instrument. Additionally, the association information containedwithin the badge may give the clinician information that allows him tochoose which medical instrument to take control of to perform a desiredprocedure. Further, the information given by the orbiting portion of thebadge may tell the clinician how to position the input control device tomatch position with the orientation of the medical instrument beforetaking control. In some embodiments, the system may require theclinician to match this position before allowing a change of state fromengagement to active.

Referring now to FIGS. 6A-6B, the distal end portion of the medicalinstrument 26 a visible in the field of view includes an end effectortip 230, a shaft 232, and a joint region 234 between the shaft and endeffector tip. The distal end portion of the medical instrument 26 cvisible in the field of view includes an end effector tip 240, a shaft242, and a joint region 244 between the shaft and end effector tip. Asshown in FIG. 6A and previously described, a default location forplacement of the badge 207 is superimposed over the joint region 234. Adefault location for placement of the badge 218 is superimposed over thejoint region 244. In FIG. 6A, the instrument distal end portions arespaced apart and easily distinguishable from each other, consequently,the badges 207 and 218 are clearly readable and clearly associated withmedical instruments 26 a and 26 c, respectively. FIG. 6B illustrates thecase where the distal end portions of instruments 26 a and 26 c areoverlapping or in very close proximity in the image 202. If allowed toremain in their default locations, the badges 207, 218 would overlap orappear adjacent or superimposed upon more than one instrument, thuscausing confusion to the clinician. As shown in FIG. 6B, to remedy thisissue, the badge 207 is moved a shift distance from its default locationnear the distal end of the joint region 234 along a longitudinal axis A1of the instrument 26 a. In this embodiment, the badge 207 is relocatedto a distal end of the shaft 232 of the instrument 26 a. Similarly, thebadge 218 is moved a shift distance from its default location near thedistal end of the joint region 244 along a longitudinal axis A2 of theinstrument 26 c. In this embodiment, the badge 218 is relocated to adistal end of the shaft 242 of the instrument 26 c. The shift distancemay correspond to the position of the badge relative to a calibratedkinematic model of the instrument as coupled to the arm and spar and/ormay correspond to the size of the instrument distal end portions in thefield of view 202. For example, the calibrated kinematic model (based,for example, on knowledge of the manipulator joint positions, kinematicsensors, size of components in the kinematic chain) provides theposition and orientation of the joint region 234 of the instrument 26 ain the surgical environment. An uncertainty factor is associated withthe determined position orientation of the joint region 234 due touncertainty in the kinematic model. Based on the kinematic model, thekinematic uncertainty factors for all of the instruments in the field ofview, and the default locations for the badges, the teleoperationalsystem may determine a proximity factor (e.g. a center-to-center oredge-to-edge distance, a percentage of overlap) indicating whether thebadges overlap or are sufficiently close to each other as to createambiguity to a clinician if the badges remain in the default locations.If the proximity factor is less than a predetermined interferencedistance, the badges are adjusted as needed to raise the proximityfactor above the predetermined interference distance. For example, insome embodiments, the badges can be moved along or across the axis ofthe instruments by at least a minimum shift distance. In otherembodiments, the badges can be resized or reshaped to maintain anacceptable interference distance while still providing the desiredinformational content. In this way, the badges 207, 218 remain clearlyassociated with their respective medical instruments 26 a, 26 c and alsoremain clearly readable.

Referring now to FIGS. 7A-7C, a badge with its association informationmay be particularly useful when a medical instrument approaches aboundary of the image 202 of the field of view or moves out of the image202. FIG. 7A illustrates the displayed image 202 of a field of view ofthe imaging instrument in the surgical environment 200. The displayedimage has a boundary 700. The distal end portion of medical instrument26 c, particularly the end effector tip 240, is within the boundary 700and is visible in the displayed image 202. The determination of whetherthe instrument 26 c is inside or outside of the field of view of theimaging instrument may be based upon the calculated location of the endeffector tips 240. Because of small error factors associated with theteleoperational system, the instrument, and/or the imaging system, thedetermination of the location of the tips 240 with respect to theimaging instrument has an associated cumulative error factor. To avoidproviding false-positive out-of-view indicators to a clinician, thedetermination of whether an instrument tip is out of the imaging systemfield of view may be biased by estimating the range of possiblelocations for the distal tip and suppressing an out-of-view indicator ifany or a designated percentage of the estimated possible locations forthe distal tip are within the field of view. The sensitivity of the biasmay be adjusted based upon the clinician's tolerance for false-positiveout-of-view indicators. A set of error bounding volumes 702 isassociated with the tip 240. The bounding volumes 702 may be displayedgraphically in the image 202 or may be associated with the tip 240without being displayed. The error bounding volumes may represent thepredicted locations of the distal and proximal ends of the tip portionsto within a high degree of certainty such as 90-99%. The error boundingvolume 702 represents the predicted locations of the tip 240 of theinstrument 26 c. Further description of the error calculation andbounding volumes is provided in U.S. Provisional App. No. 61/954,442,filed Mar. 17, 2014, disclosing “Systems and methods for offscreenindication of instruments in a teleoperational medical system,” which isincorporated by reference herein in its entirety.

As shown in FIG. 7A, the bounding volumes 702 are within the field ofview bounded by the boundary 700. The bounding volumes 702 may or maynot be rendered in the image 202, but their location relative to theboundary may be determined by the system regardless of whether they aredisplayed. Since the medical instrument 26 c is visible in the displayand the bounding volumes are within the boundary 700, the badge 218 isaccordingly displayed on the image 202 of the field of view. In order tomaintain visibility and readability, the badge 218 remains completelyinside the boundary 700, even if the distal end portion of theinstrument 26 c, to which the badge 206 is attached, is partiallyoutside the boundary 700, as shown here. This allows the clinician toclearly locate the medical instrument 26 c even if it is only barelyvisible within the image 202. In FIG. 7B the medical instrument 26 c isnot visible in the displayed image 202. In some embodiments, the systemmay terminate display of the badge 218 when the associated medicalinstrument 26 c is not visible in the image 202. Because the medicalinstrument 204 may be teleoperationally controlled without being visiblein the field of view to a clinician, inadvertent movement of aninstrument outside of the field of view creates a safety risk.Additionally, clinicians may lose track of instruments that are locatedoutside of the field of view. To minimize these risks, out-of-viewinstrument indicators may be visually or audibly presented to increasethe clinician's awareness of the location of instruments not visiblewithin the field of view. However, in FIG. 7B the bounding volumes 702for medical instrument 26 c are within the boundary 700 indicating thatthe location of the tip 240 relative to the boundary 700 is uncertain.Therefore, while the bounding volumes 702 remain within the boundary700, the badge 218 remains on the image 202 even if the image 202 doesnot appear to include the tip 240.

Once the medical instrument 204 is far enough outside the boundary 700that the bounding volumes 702 are also outside of the boundary, anout-of-view instrument indicator 704 is provided along the boundary 700of the image 202 of the field of view, as shown in FIG. 7C, to indicatethat medical instrument 26 c is located out of the field of view in thegeneral direction of the indicator. In this embodiment, the indicator704 is a graphical bar, but in other embodiments may be a series ofdots, or an icon, an alpha-numeric indicator. In addition to oralternative to the visual indicator 704, an audible out-of-viewindicator such as a beeping sound or a language-based instruction mayalert the clinician that the medical instrument 204 is out of the fieldof view. The audible cue may pan between left and right speakers of thesurgeon's console to reinforce the instrument position relative to theview. Alternatively, the audible cue may emit from the left or rightspeaker in correspondence with the left or right hand control associatedwith the instrument. In addition to or alternative to the visualindicator 704, textual information 706 related to the out-of viewinstrument may be provided to alert the clinician and/or to provideidentifying information about the instrument or an instruction tovisualize the instrument. In various embodiments, the out-of-viewindicator 704 is mutually exclusive with the badge 218. When the badge218 is displayed the out-of-view indicator 704 is not displayed, andvice versa. This provides the clinician with constant knowledge of thelocation of the medical instrument 26 c.

In various embodiments, the use of an out-of-view indicator may belimited to avoid becoming a distraction to the clinician. The use of theout-of-view indicator may be context-sensitive such that the out-of-viewindicator may only be displayed during certain states of operation ofthe teleoperational system. For example, the out-of-view indicator maybe displayed during a state of the system in which the operator controlsmovement of the imaging system, a state which may be known as a cameracontrol state. As another example, the out-of-view indicator may bedisplayed while the system awaits input from the operator to takecontrol of an associated instrument, described above as the disengagedstate. As another example, the out-of-view indicator may be displayedfor a few seconds after initiating a state of the system in which theoperator controls movement of the instruments, described above as theengaged state. In still other alternative embodiments, the out-of-viewindicator may be disabled or selectively enabled when the clinicianwants to learn about the location of out-of-view instruments. In someembodiments, the clinician must provide an acknowledgement that theinstrument tip is outside of the field of view before operation of theout of view instrument is enabled. Additional warnings oracknowledgements may be used for energy emitting devices, sharp devices,or devices that provide an increased patient risk if used withoutvisualization. Further description of an out-of-view indicator system isprovided in U.S. Provisional App. No. 61/954,442 which is incorporatedby reference above.

As explained above, the badges may be positioned and sized to correspondwith the images of the instrument distal ends with which they areassociated. FIGS. 8-11 provide further detail for preserving thisphysical correspondence. FIG. 8 illustrates a method 800 of renderingbadges within image 202. At process 802 the locations of medicalinstrument distal end portions are projected into a coordinate space ofthe image 202. FIG. 9 illustrates a method 900 for performing theprocess 802. At process 902, the forward kinematics of theteleoperational assembly are evaluated to determine the position andorientation of the medical instruments (including the imaging device)within a world coordinate system (e.g., a surgical environmentcoordinate space F_(W)). More specifically, the kinematics associatedwith the respective set of links and teleoperational manipulator foreach medical instrument is determined in the world coordinate system.The world coordinate system is a set of Cartesian coordinates that isestablished with reference to the surgical environment within which theteleoperational system is located.

At process 904 the position and orientation of the distal end portion ofeach medical instrument (e.g. in the instrument coordinate space F_(I))is mapped to an endoscope coordinate system (e.g. imaging devicecoordinate space F_(E)). The endoscope coordinate system is a set ofCartesian coordinates that is established with its origin at the distaltip of the endoscope. By using the forward kinematics of the endoscopeand the forward kinematics of each medical instrument in the worldcoordinate system, as determined at process 902, the medical instrumentdistal end position and orientation relative to the endoscope tipposition and orientation may be computed in the endoscope coordinatesystem. Generally, the default location for the badges may be at or neara kinematic endpoint of the medical instrument.

At process 906 a calibrated camera model is obtained for the endoscope.The calibrated camera model is a mapping from endoscope coordinate spaceto the right and left eye image coordinate spaces. In one embodiment,the mapping is accomplished with a series of three transforms. First, amodelview transform maps from the endoscope coordinate space (at thedistal tip of the endoscope) into an eye coordinate space for theendoscope, accounting for interocular separation of the stereoscopiccameras in the imaging system and for differences in coordinate systemconventions. Second, a perspective projection transform maps from theeye coordinate space to normalized coordinates (e.g., −1, +1) andaccounts for perspective of the endoscope field of view and convergencedistance. Third, a viewport bounds transform maps from the normalizedeye coordinates to the right and left eye image coordinate spaces. Inone embodiment, these three transforms can be combined into one 4×4homogenous transform for each of the right and left eye images.

At process 908 the calibrated camera model is used to project points andvectors from endoscope coordinate space to the image coordinate spacefor each eye. More specifically the calibrated camera model projects themedical instrument distal end position and orientation to a right eyeimage coordinate space and a left eye image coordinate space to providea stereoscopic 3D image to the clinician.

At process 910, uncertainties in the kinematic model of theteleoperational system, which result in corresponding uncertainties inthe position of the distal end portion of the associated medicalinstrument, are estimated with respect to the endoscope tip. At process912 these uncertainties are mapped to the right and left eye imagecoordinate spaces from which the clinician views the three dimensionalimage 202.

At process 914, using the forward kinematics determined at process 902,the kinematic remote center (e.g., the incision site about which theinstruments pivot) is mapped to the endoscope coordinate space. Atprocess 916, the calibrated camera model is used to project the remotecenter to the image coordinate systems for each eye in a similar manneras described in process 908.

The processes 902-916 may be repeated for each medical instrument in theteleoperational system so that all medical instruments are properlyprojected into the stereoscopic 3D image 202. The processes may beperformed in sequence or in parallel, and one or more process may beupdated while the others are not.

Referring again to FIG. 8, the rendering of badge graphics is based upona consideration of multiple display factors including, for example, thedefault locations at or near the distal tip of the distal ends of themedical instruments, the potential ambiguity associated with overlappingor indeterminate badge placement, the constraints imposed by theendoscope viewport, and the system rules for hidden instruments. Theposition of the badge graphics may be determined in image coordinatespace for each eye. At a process 804, potential overlap of badges ischecked for and resolved to reduce the likelihood that a badge isspatially associated with the wrong instrument. FIG. 10 illustrates amethod 1000 for performing the process 804 of FIG. 8. In thisembodiment, at least two medical instruments are positioned in thesurgical space. As described above, the default location for renderingthe badge is at the distal end portion of a medical instrument. Atprocess 1002, the distance in image space coordinates between the distalend portions of two medical instruments is calculated. At process 1004,a threshold for minimum allowed inter-badge distance is calculated. Thethreshold for minimum allowed inter-badge distance is the maximum of thesum of the projected radii (or, if the badge is not circular, anothermaximum dimension from the center of the badge) of the badges plus themaximum of the sum of the projected uncertainty of badge positions. Atprocess 1006, the image space distance between the two distal endportions is compared to the calculated threshold distance to determinewhether their associated badges overlap or may be perceived to overlap.If overlap is detected, at process 1008 a direction to move the badge toresolve the overlap is computed. The direction to move the badge isdetermined by the unit vector between the instrument distal end portionand the remote center projected into image coordinate space (asdetermined in processes 908 and 916). Further if overlap is detected, atprocess 1010 a distance to translate the badge to resolve overlap isdetermined. The distance that the badges are offset is computed as thedifference between the minimum allowed inter-badge distance and thecurrent distance between the distal end portions. The offset preventsbadge graphics from overlapping and provides badge positions that aresufficiently spaced to avoid uncertainty in instrument association.

Referring again to FIG. 8, at a process 806 badges are constrained tothe viewable space of image 202, which corresponds to the boundaries ofthe endoscope viewport at the distal tip of the endoscope. This processmay place the badge graphics within the boundaries of the image 202without clipping any of the graphics. This process may also control theperceived depth of the badge by preventing modification to thehorizontal disparity if the badges are shifted to avoid the imageboundaries or to prevent ambiguity. FIG. 11 illustrates a method 1100for performing the process 806 of FIG. 8. At process 1102, the right eyeboundary of the viewable space of image 202, corresponding to theendoscope viewport, is determined. At process 1104 the projected radiusof the badge graphic is determined for the right eye. At process 1106,for the right eye, the boundary is inset by the projected badge radiusto create an inset boundary. At process 1108 the Y position of thecenter of the badge is adjusted to remain within the inset boundary forthe right eye. At process 1110 the offset of the X position of thecenter of the badge which is necessary to keep the badge completelywithin the inset boundary is computed. Processes 1102 through 1110 arerepeated for the left eye images. To maintain the proper apparent depthin stereoscopic 3D, the badge must be adjusted the same amount in boththe right and left eye images. This maintains the horizontal disparityrequired for the proper perceived depth of the badge. At process 1112,the X offset which was computed at process 1110 for one of the right orleft eye images is applied to the X position of the badge in both theright and left eye images. In some embodiments, the larger X offsetvalue is chosen to prevent the possibility that the badge graphic willbe even partially outside of the boundary 700.

Referring again to FIG. 8, at process 808, the badges are renderedwithin the left and right eye displays to form the image 202 based onthe calculations made in processes 802, 804 and 806. One or moreelements in embodiments of the invention may be implemented in softwareto execute on a processor of a computer system such as controlprocessing system. When implemented in software, the elements of theembodiments of the invention are essentially the code segments toperform the necessary tasks. The program or code segments can be storedin a processor readable storage medium or device that may have beendownloaded by way of a computer data signal embodied in a carrier waveover a transmission medium or a communication link. The processorreadable storage device may include any medium that can storeinformation including an optical medium, semiconductor medium, andmagnetic medium. Processor readable storage device examples include anelectronic circuit; a semiconductor device, a semiconductor memorydevice, a read only memory (ROM), a flash memory, an erasableprogrammable read only memory (EPROM); a floppy diskette, a CD-ROM, anoptical disk, a hard disk, or other storage device, The code segmentsmay be downloaded via computer networks such as the Internet, Intranet,etc.

Note that the processes and displays presented may not inherently berelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct a morespecialized apparatus to perform the operations described. The requiredstructure for a variety of these systems will appear as elements in theclaims. In addition, the embodiments of the invention are not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may occur to those ordinarily skilled in theart.

1-20. (canceled)
 21. A system comprising: a teleoperational assemblyincluding a first manipulator configured to control movement of a firstmedical instrument; and a processing unit including one or moreprocessors, wherein the processing unit is configured to: display animage of a field of view of a surgical environment; display a firstbadge at a default location relative to an image of the first medicalinstrument in the image of the field of view, the first badge includingfirst association information for the first medical instrument;determine a display factor associated with the first badge; responsiveto the display factor, determine a shift distance between the defaultlocation and a shifted location; and display the first badge at theshifted location.
 22. The system of claim 21, wherein the display factoris an overlap of the first badge with a second badge displayed in theimage of the field of view.
 23. The system of claim 21, wherein thedisplay factor is an overlap of the first badge with an image of asecond medical instrument in the image of the field of view.
 24. Thesystem of claim 21, wherein the display factor is a viewport constraintfor an imaging instrument capturing the field of view of the medicalinstrument.
 25. The system of claim 21, wherein the display factor is anobstruction of the image of the first medical instrument in the image ofthe field of view.
 26. The system of claim 21, wherein the shiftdistance is in a linear direction aligned with a shaft of the firstmedical instrument.
 27. The system of claim 21, wherein the shiftdistance is based on a threshold interference distance between the firstbadge and a second badge displayed in the image of the field of view.28. The system of claim 21, wherein the processing unit is furtherconfigured to: responsive to the display factor, determine a revisedsize for the first badge and display the first badge using the revisedsize.
 29. The system of claim 21, wherein the processing unit is furtherconfigured to: responsive to the display factor, determine a revisedshape for the first badge and display the first badge using the revisedshape.
 30. The system of claim 21, wherein the first badge includes astatic central portion and an orbiting portion configured to movecircumferentially around the static central portion.
 31. The system ofclaim 21, wherein the teleoperational assembly further includes a secondmanipulator configured to control movement of a second medicalinstrument; wherein the processing unit is further configured to displaya second badge proximate to an image of the second medical instrument inthe image of the field of view, the second badge including secondassociation information for the second medical instrument; whereindetermining the display factor includes determining a distance betweenthe first and second badges and recognizing that the determined distanceis less than a predetermined interference distance; and wherein theshift distance is in a linear direction aligned with a shaft of thefirst medical instrument.
 32. The system of claim 31, wherein the firstassociation information includes information about the firstmanipulator, and wherein the second association information includesinformation about the second manipulator.
 33. The system of claim 31,wherein the processing unit is further configured to shift the secondbadge in a linear direction aligned with a shaft of the second medicalinstrument if the determined distance is less than the predeterminedinterference distance.
 34. The system of claim 31, wherein determiningthe distance between the first and second badges includes determining aproximity factor corresponding to the first and second badges.
 35. Thesystem of claim 34, wherein determining the proximity factor includesdetermining a distance between a center of the first badge and a centerof the second badge.
 36. The system of claim 34, wherein determining theproximity factor includes determining a distance between an edge of thefirst badge and an edge of the second badge.
 37. The system of claim 34,wherein determining the proximity factor includes determining apercentage of overlap between the first and second badges.
 38. Thesystem of claim 31, wherein the processing unit is further configured toadjust a size of the first badge if the determined distance is less thanthe predetermined interference distance.
 39. The system of claim 31,wherein the processing unit is further configured to adjust a shape ofthe first badge if the determined distance is less than thepredetermined interference distance.
 40. The system of claim 31, whereindisplaying the first badge includes displaying the first badgesuperimposed on an image of a distal end portion of the first medicalinstrument, and wherein displaying the second badge includes displayingthe second badge superimposed on an image of a distal end portion of thesecond medical instrument.